Papers with pretrained BERT model
Understanding BERT performance in propaganda analysis (D19-50)
Copied to clipboard
| Challenge: | Despite the challenging nature of the shared task, our pretrained BERT model scored 0.62 F1 on the test set and ranked third among 25 teams who participated in the contest. |
| Approach: | They propose to use a dataset to fine-tune a model for propaganda analysis at sentence level to determine whether a text is 'propaganda' and to examine false-positive cases. |
| Outcome: | The proposed model scored 0.62 F1 on the test set and ranked third among 25 teams who participated in the shared task. |
E-BERT: Efficient-Yet-Effective Entity Embeddings for BERT (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to enhance BERT with factual knowledge about entities require no additional pretraining and no changes to the encoder itself. |
| Approach: | They propose a way to inject factual knowledge into the pretrained BERT model by aligning Wikipedia2Vec entity vectors with BERT's native wordpiece vector space and feeding the aligned entity vector into BERT as if they were wordpieces. |
| Outcome: | The proposed version outperforms baseline models on unsupervised question answering, supervised relation classification and entity linking tasks. |
Japanese Zero Anaphora Resolution Can Benefit from Parallel Texts Through Neural Transfer Learning (2021.findings-emnlp)
Copied to clipboard
| Challenge: | Using a pretraining model, we find that the performance of Japanese zero anaphora resolution (ZAR) is improved by using machine translation. |
| Approach: | They propose to inject machine translation as an intermediate task between pretraining and ZAR by injecting machine translation into a pretrained BERT model and injecting it into MT. |
| Outcome: | The proposed framework shows that Japanese zero anaphora resolution (ZAR) can be improved by transfer learning from machine translation (MT). |
Finding Friends and Flipping Frenemies: Automatic Paraphrase Dataset Augmentation Using Graph Theory (2020.findings-emnlp)
Copied to clipboard
| Challenge: | Having high quality annotated data is crucial for training supervised machine learning models. |
| Approach: | They propose automated methods to improve NLP datasets by viewing them as graphs with expected semantic properties. |
| Outcome: | The proposed methods improve paraphrase models on pre-trained datasets. |
HyperBERT: Mixing Hypergraph-Aware Layers with Language Models for Node Classification on Text-Attributed Hypergraphs (2024.findings-emnlp)
Copied to clipboard
| Challenge: | Existing methods to learn informative data representations on text-attributed hypergraphs struggle to capture full extent of hypergraph structural information and rich linguistic attributes inherent in the nodes attributes. |
| Approach: | They propose to augment a pre-trained BERT model with specialized hypergraph-aware layers for the task of node classification. |
| Outcome: | The proposed model outperforms existing methods on five challenging text-attributed hypergraph node classification benchmarks. |